Implicit Interaction with Textual Information using Physiological Signals

Implicit interaction refers to human-computer interaction techniques that do not require active engagement from the users. Instead, the user is passively monitored while performing a computer task, and the data gathered is used to infer implicit measures as inputs to the system. Among the multiple applications for implicit interaction, collecting user feedback on information content is one that has increasingly been investigated. As the amount of available information increases, traditional methods that rely on the users' explicit input become less feasible. As measurement devices become less intrusive, physiological signals arise as a valid approach for generating implicit measures when users interact with information. These signals have mostly been investigated in response to audio-visual content, while it is still unclear how to use physiological signals for implicit interaction with textual information. This dissertation contributes to the body of knowledge by studying physiological signals for implicit interaction with textual information. The research targets three main research areas: a) physiology for implicit relevance measures, b) physiology for implicit affect measures, and c) physiology for real-time implicit interaction. Together, these provide understanding not only on what type of implicit measures can be extracted from physiological signals from users interacting with textual information, but also on how these can be used in real time as part of fully integrated interactive information systems. The first research area targets perceived relevance, as the most noteworthy underlying property regarding the user interaction with information items. Two experimental studies are presented that evaluate the potential for brain activity, electrodermal activity, and facial muscle activity as candidate measures to infer relevance from textual information. The second research area targets affective reactions of the users. The thesis presents two experimental studies that target brain activity, electrodermal activity, and cardiovascular activity to indicate users' affective responses to textual information. The third research area focuses on demonstrating how these measures can be used in a closed interactive loop. The dissertation reports on two systems that use physiological signals to generate implicit measures that capture the user's responses to textual information. The systems demonstrate real-time generation of implicit physiological measures, as well as information recommendation on the basis of implicit physiological measures. This thesis advances the understanding of how physiological signals can be implemented for implicit interaction in information systems. The work calls for researchers and practitioners to consider the use of physiological signals as implicit inputs for improved information delivery and personalization.Implisiittinen vuorovaikutus viittaa ihmisen ja tietokoneen välisen vuorovaikutuksen tekniikoihin, jotka eivät vaadi käyttäjän tarkkaavaisuutta. Tämän sijaan järjestelmä kerää käyttäjästä tietoja passiivisesti ja käyttää näitä tietoja operatiivisina syötteinä. Esimerkiksi viestiä kirjoitettaessa (eksplisiittinen vuorovaikutus) järjestelmä tunnistaa tekemämme kirjoitusvirheen ja automaattisesti korjaa väärin kirjoitetun sanan (implisiittinen vuorovaikutus). Implisiittinen vuorovaikutus mahdollistaa näin uusia vuorovaikutuskanavia vaivaamatta lainkaan käyttäjää. Mittauslaitteiden kehityksen myötä implisiittisessä vuorovaikutuksessa voidaan hyödyntää myös fysiologisia signaaleja, kuten aivovasteita ja kardiovaskulaarisia reaktioita. Näiden signaalien analyysi paljastaa tietoja käyttäjän kiinnostuksen kohteista ja tunteista suhteessa tietokoneen esittämään sisältöön, ja tarjoaa näin järjestelmälle paremmat mahdollisuudet vastata käyttäjän tarpeisiin. Väitöskirjani tarkoituksena on tutkia käyttäjien fysiologisia signaaleja sekä kerätä tietoa heidän reaktioistaan ja mielipiteistään suhteessa tekstipohjaiseen informaatioon ja käyttää näitä signaaleja ja tietoja implisiittisen vuorovaikutuksen mahdollistamiseksi. Tarkkaan ottaen tarkoituksenani on tutkia a) fysiologisten signaalien kykyä kertoa siitä, miten kiinnostavana käyttäjä kokee lukemansa tekstin, b) fysiologisten signaalinen käyttökelpoisuutta ennustamaan, minkälaisia tunnereaktiota (esim. huvittuneisuutta) tekstit herättävät lukijassa sekä, c) fysiologisen signaalien käyttökelpoisuutta reaaliaikaisessa implisiittisessä vuorovaikutuksessa. Tutkimuksen tulokset osoittavat, että fysiologiset signaalit tarjoavat toimivan ratkaisun reaaliaikaiseen implisiittiseen vuorovaikutukseen tekstipohjaisten sisältöjen parissa. Tutkimuksen löydösten pääviesti tutkimusyhteisölle ja alan ammattilaisille on se, että implisiittisinä syötteinä fysiologiset signaalit helpottavat informaation kulkua ja parantavat personalisoimista ihmisen ja tietokoneen välisessä vuorovaikutuksessa

Exploring the applicability of implicit relevance measures in varying reading speed for adaptive I.R. systems

This thesis goes further in the study of implicit indicators used to infer interest in documents for information retrieval tasks. We study the behavior of two different categories of implicit indicators: fixation-derived features (number of fixations, average time of fixations, regression ratio, length of forward saccades), and physiology (pupil dilation, electrodermal activity). Based on the limited number of participants at our disposal we study how these measures react when addressing documents at three different reading rates. Most of the fixation-derived features are reported to differ significantly when reading at different speeds. Furthermore, the ability of pupil size and electrodermal activity to indicate perceived relevance is found intrinsically dependent on speed of reading. That is, when users read at comfortable reading speed, these measures are found to be able to correctly discriminate relevance judgments, but fail when increasing the addressed speed of reading. Therefore, the outcomes of this thesis strongly suggest to take into account reading speed when designing highly adaptive information retrieval systems

Exploring the applicability of implicit relevance measures in varying reading speed for adaptive I.R. systems

Projecte realitzat en el marc d’un programa de mobilitat amb la University of Helsinki. Faculty of Science. Department of Computer ScienceThis thesis goes further in the study of implicit indicators used to infer interest in documents for information retrieval tasks. We study the behavior of two different categories of implicit indicators: fixation-derived features and physiology (pupil size, electrodermal activity). Based on the limited number of participants at our disposal we study how these measures react when addressing documents at three different reading rates. Most of the fixation-derived features are reported to differ significantly when reading at different speeds. Furthermore, the ability of pupil size and electrodermal activity to indicate perceived relevance is found intrinsically dependent on speed of reading. That is, when users read at comfortable reading speed, these measures are found to be able to correctly discriminate relevance judgments, but fail when increasing the addressed speed of reading. Therefore, the outcomes of this thesis strongly suggest to take into account reading speed when designing highly adaptive information retrieval systems

Motivational intensity and visual word search : Layout matters

Motivational intensity has been previously linked to information processing. In particular, it has been argued that affects which are high in motivational intensity tend to narrow cognitive scope. A similar effect has been attributed to negative affect, which has been linked to narrowing of cognitive scope. In this paper, we investigated how these phenomena manifest themselves during visual word search. We conducted three studies in which participants were instructed to perform word category identification. We manipulated motivational intensity by controlling reward expectations and affect via reward outcomes. Importantly, we altered visual search paradigms, assessing the effects of affective manipulations as modulated by information arrangement. We recorded multiple physiological signals (EEG, EDA, ECG and eye tracking) to assess whether motivational states can be predicted by physiology. Across the three studies, we found that high motivational intensity narrowed visual attentional scope by altering visual search strategies, especially when information was displayed sparsely. Instead, when information was vertically listed, approach-directed motivational intensity appeared to improve memory encoding. We also observed that physiology, in particular eye tracking, may be used to detect biases induced by motivational intensity, especially when information is sparsely organised.Peer reviewe

Exploring Peripheral Physiology as a Predictor of Perceived Relevance in Information Retrieval

Peripheral physiological signals, as obtained using electrodermal activity and facial electromyography over the corrugator supercilii muscle, are explored as indicators of perceived relevance in information retrieval tasks. An experiment with 40 participants is reported, in which these physiological signals are recorded while participants perform information retrieval tasks. Appropriate feature engineering is defined, and the feature space is explored. The results indicate that features in the window of 4 to 6 seconds after the relevance judgment for electrodermal activity, and from 1 second before to 2 seconds after the relevance judgment for corrugator supercilii activity, are associated with the users’ perceived relevance of information items. A classifier verified the predictive power of the features and showed up to 14% improvement predicting relevance. Our research can help the design of intelligent user interfaces for information retrieval that can detect the user’s perceived relevance from physiological signals and complement or replace conventional relevance feedback

Natural brain-information interfaces : Recommending information by relevance inferred from human brain signals

Peer reviewe

Predicting term-relevance from brain signals (Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval)

Term-Relevance Prediction from Brain Signals (TRPB) is proposed to automatically detect relevance of text information directly from brain signals. An experiment with forty participants was conducted to record neural activity of participants while providing relevance judgments to text stimuli for a given topic. High-precision scientific equipment was used to quantify neural activity across 32 electroencephalography (EEG) channels. A classifier based on a multi-view EEG feature representation showed improvement up to 17% in relevance prediction based on brain signals alone. Relevance was also associated with brain activity with significant changes in certain brain areas. Consequently, TRPB is based on changes identified in specific brain areas and does not require user-specific training or calibration. Hence, relevance predictions can be conducted for unseen content and unseen participants. As an application of TRPB we demonstrate a high-precision variant of the classifier that constructs sets of relevant terms for a given unknown topic of interest. Our research shows that detecting relevance from brain signals is possible and allows the acquisition of relevance judgments without a need to observe any other user interaction. This suggests that TRPB could be used in combination or as an alternative for conventional implicit feedback signals, such as dwell time or click-through activity

BCI for physiological text annotation

IUI'17Peer reviewe

Extracting Relevance and Affect Information from Physiological Text Annotation

We present physiological text annotation, which refers to the practice of associating physiological responses to text content in order to infer characteristics of the user information needs and affective responses. Text annotation is a laborious task, and implicit feedback has been studied as a way to collect annotations without requiring any explicit action from the user. Previous work has explored behavioral signals, such as clicks or dwell time to automatically infer annotations, and physiological signals have mostly been explored for image or video content. We report on two experiments in which physiological text annotation is studied first to 1) indicate perceived relevance and then to 2) indicate affective responses of the users. The first experiment tackles the user’s perception of relevance of an information item, which is fundamental towards revealing the user’s information needs. The second experiment is then aimed at revealing the user’s affective responses towards a -relevant- text document. Results show that physiological user signals are associated with relevance and affect. In particular, electrodermal activity (EDA) was found to be different when users read relevant content than when they read irrelevant content and was found to be lower when reading texts with negative emotional content than when reading texts with neutral content. Together, the experiments show that physiological text annotation can provide valuable implicit inputs for personalized systems. We discuss how our findings help design personalized systems that can annotate digital content using human physiology without the need for any explicit user interaction